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Abstract 

An extension of the Gauss-Newton algorithm is proposed to find local minimizers of pe- 
nalized nonlinear least squares problems, under generalized Lipschitz assumptions. Conver- 
gence results of local type are obtained, as well as an estimate of the radius of the convergence 
ball. Some applications for solving constrained nonlinear equations are discussed and the nu- 
merical performance of the method is assessed on some significant test problems. 
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1 Introduction 

Given Hilbert spaces X and Y , a Frechet differentiable nenlinear eperater F .X , and a cenvex 
lewer semicentinueus penalty functional / : X — )• MU {+oo}, we consider the optimization problem 

mm-\\Fix)-yf +Jix):=^(x). (^) 
xex 2 

Problem (^) is in general a nonconvex and nonsmooth problem, having on the other hand a 
particular structure: it is in fact the sum of a nonconvex, smooth term and a convex and possibly 
nonsmooth one. The aim of the paper is to find a convergent algorithm towards a local minimizer 
of <I>, assuming that it exists. Motivated by several applications lfT3l l40l [T6l . problem (^) is 
receiving an increasing attention. In particular, for 7 = 0, (^) is a classical nonlinear least squares 
problem llT2l |49l . This kind of problems can be solved by general optimization methods, but 
typically is solved by more efficient ad hoc methods. In many cases they achieve better than linear 
convergence, sometimes even quadratic, even though they do not need computation of second 
derivatives. Among the various approaches, one of the most popular is the Gauss-Newton method, 
introduced in |i7j| : 

Xn+l =Xn- \F'{XnTF'{Xn)]-'F'{XnT{F{Xn) -j). (1) 
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Under suitable assumptions, such a procedure is convergent to a stationary point of ;c i— ^ 1 1 F (x) — j | 
namely to a point x such that F'{x)*{F{x) —y) = 0. Moreover, one can easily show that the point 
Xn^i defined in ([T]) is the minimizer of the "linearized" functional: 

xh^ ^\\F{xn) + F'{x„){x-x„) -y\\'^. (2) 

There is a wide literature devoted to the study of convergence results for the Gauss-Newton method 
under different perspectives. In particular we can distinguish two main streams of research: the 
papers devoted to a local analysis, and the ones devoted to semilocal results. The first class of 
studies fOl [U 121 |3ll assume the existence of a local minimizer, and they actually determine 
a region of attraction around that point, meaning that if the starting point is chosen inside that 
region the iterative process is guaranteed to converge towards the minimizer On the contrary, the 
semilocal results — also known as Kantorovich type theorems — do not assume the existence of a 
local minimizer, they just establish sufficient conditions on the starting point in order to make the 
iterative procedure convergent towards a point that is proved to be a local minimizer [3l [T4l[T8ll27l . 

In this paper we propose a generalization of the Gauss-Newton algorithm to the case in which 
/ 7^ 0, that reads as follows: 

X„+i = proxf {Xn - [F'{XnTF'{Xn)]-'F'{Xnr{F{Xn) -j)) , (3) 

where prox^^^"^ is the proximity operator associated to / (see 1^331 [34l [35l ). with respect to the 
metric defined by the operator H{x„) := F'(x„)*F'(x„). The algorithmic framework in (|3]l is de- 
termined following the same line of (|2]l, i.e. linearizing the functional F at the point x„, and 
computing the minimizer of the corresponding "linearized" functional 

2\\^(^")+^ (x„)(x-x„) -y\\ +7(x) 

This approach is the common way to deal with generalizations of the Gauss-Newton method, as 
we better explain in the next section. The convergence results we obtain are of local type, and 
they are comparable to those obtained for the classical Gauss-Newton method. In particular, we 
get linear convergence in the general case, and quadratic convergence for zero residual problems. 
Furthermore, we are able to give an estimate of the radius of the convergence ball around a local 
minimizer. It should be noted that the computation of the proximity operator is in general not 
straightforward and it may require an iterative algorithm itself, since in general a closed form is 
not available. On the other hand, we could have denoted x„+i simply as the minimizer of the 
generalized version of (|2]i. The formulation in terms of proximity operators allows to use the well 
developed theory on this kind of operators, and in our opinion enlightens the connections and the 
differences with other first order methods that have recently been proposed to solve problem (^) 
(see the next section for further details). 

The paper is organized as follows: we start with an analysis of the state-of-the art literature 
on related problems in Section |2l and then in Section |3] we review the basic concepts that will be 
used: generalized Lipschitz conditions, generalized inverses and proximity operators. In Section 
HI the minimization problem is precisely stated and some necessary conditions satisfied by local 
minimizers are presented. The main result of the paper. Theorem [T] is discussed in Section |5] 
and proved in Section |6] Section [7] gives an application to the problem of constrained nonlinear 
equations and, finally, in Section [8] numerical tests are set up and analyzed. 
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2 Comparison with related work 



We review the available algorithms to solve this problem, enlightening the connections and the 
differences with our approach. 

Convex composite optimization Problem (^) can be cast, in principle, as a composite opti- 
mization problem of the form 

mmh{c{x)), (4) 
by setting h:Y xX^RU {+0°} and c : X ^ Y x X as follows: 

h{s,v) = \\sf+J{v), c{x) = {F{x)-y,x). (5) 

Problem © has been deeply studied in HOl HH |28l |29l |47l |48l from different point of views, 
mainly in the case X = W and Y = R'", but the hypotheses significantly differ, as well as the 
obtained results. More specifically, in ll26l . the assumptions are too general to capture the features 
of problem (^), and allow only to get convergence results much weaker than the ones obtained 
in Theorem [T] Regarding all the remaining papers, as a matter of fact, the following special case 
of inclusion problem is treated 

c{x)^C, C = argmin/j. (6) 

In particular, the existence of an x such that c{x) G argmin/j is always assumed. That hypothesis is 
of course reasonable if we think of as a kind of norm, but if we take it as in Q, then C = {{s,v) : 
5 = 0,v G argmin/} and we are lead to the condition 

3x G X, F{x) = y and x G argmin/ 

which is too demanding for our original problem (^). 

Nonlinear inverse problems with regularization Here the problem is to solve the nonlinear 
equation 

F{x)=y (7) 

in the ill-posed case. Typically a solution is found by introducing a regularization term weighted 
with a positive parameter. There are two possible approaches. The first one employs iterative 
methods which deal directly with problem see 14 ]. In this case an iterative process is set up by 
minimizing at each step a simplified regularized problem (generally linear) having the structure of 
(^) — with a weight for J varying at each iteration. Within this class of methods, one popular 
choice is the iteratively regularized Gauss-Newton method, see ||8l|20l|2ll|25|. Anyway, we remark 
that, despite the name, that algorithm is different from any kind of Gauss-Newton optimization 
method above, since it is not designed to "optimize" any objective functional but, in fact, it 
directly looks for an exact solution of the equation (|7]l. 

An alternative approach is the classical Tikhonov method |[T3l replacing dT]) by the minimiza- 
tion of the associated Tikhonov functional. A problem of the same type of (^) arises, with J 
weighted by a properly chosen parameter. Up to our knowledge, besides our method, the papers 
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by Ramlau and Teschke, e.g. Il37l[38l . are the only ones providing algorithms for the minimiza- 
tion of such type of functionals. In addition, the scheme they propose is different from ours and in 
general it converges only weakly. On the other hand, our algorithm assumes the derivative F'{x) 
to be injective with closed range — a hypothesis not suitable for handling the ill-posed case. 

3 Preliminaries and notations 

We start with some notations. In the whole paper X and Y denote a Hilbert space and H is a 
nonempty open subset of X. F : Q. C X ^ Y is an operator, in general nonlinear, and F' denotes 
its derivative (Frechet or Gateaux). L(X,F) is the space of bounded linear operators from X to Y. 
If A € L(X,F), RiA) and N{A) shall denote respectively its range and kernel and A* its adjoint. 

3.1 Generalized Lipschitz conditions 

Convergence results for the Gauss-Newton algorithm are typically obtained requiring Lipschitz 
continuity of the operator F' In B4ll45l . Wang introduced some weaker notions of Lipschitz 
continuity in the context of Newton's method. Recently, also convergence of the Gauss-Newton 
method has been proved under such generalized Lipschitz conditions, see e.g. |[3T1 [T] |2l. In 
this section we recall the definitions and review some basic properties of generalized Lipschitz 
continuous functions. 

First of all recall that a set C X is called star shaped with respect to some of its points G f/ 
if the segment [x^,x\ is contained in U for every x ^U. 

Definition 1. Let / : 11 C X — 7 and ?7 C H be a starshaped set with respect to x^ £ U. Fix 
R G (0,+oo] such that sup^^y ||x — x*|| < R and let L : [0,R) — M be a positive and continuous 
function. The mapping / is said to satisfy the radius Lipschitz condition of center x^ with L 
average on U if 



for all t G [0, 1] and xeU. 

Note that denoting by F : [0,R) — )• M a primitive function of L, e.g. r{u) = L{v) dv, inequal- 
ity dH]) can be written as 



By definition F is absolutely continuous and differentiable, with F'(m) = L(m). Since L > 0, F is 
monotone increasing. Assuming L to be increasing, we get that F is convex. 

Definition 2. Assume the hypotheses of Definition [T] We say that f : Q. C X ^Y satisfies the 
center Lipschitz condition of center with L average on U if it verifies 




(8) 



\\f{x)-f{x,+t{x-x,))\\ < F(||x-x,||) -F(f||x-x,||). 




(9) 



for every x eU. 
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The original definitions of Wang P4l do not require tlie continuity of tlie function L but just an 
integrability assumption. Our choice simplifies the proofs, but we remark that this requirement is 
not essential. Indeed, the proofs can be modified to handle also the more general case of Wang, at 
the cost of slight technical complications. In addition, the most well-known examples of Lipschitz 
averages are continuous. 

Remark 1, If / is Lipschitz continuous on a convex set U with constant L then it satisfies the 
radius and center Lipschitz condition of center with average constantly equal to L on U , for 
every x^^U . Vice versa, in the definitions above, if we take L constant we obtain two intermediate 
concepts of Lipschitz continuity, called radius and center Lipschitz continuity, which are still in 
general weaker than the classical notion, being centered at a specific point. 

Note also that the center Lipschitz condition is weaker than the corresponding radius one. 

Let F:QCX— )>Fbeaa Frechet differentiable operator. It is well-known, see e.g. |[36) . that if 
F' is Lipschitz continuous with constant L then the following inequality holds for every x,x^ G X: 



We are going to show that under the weaker Lipschitz conditions introduced above it is still pos- 
sible to prove two estimates which are similar to ([10) . To this aim we need the following three 
propositions. In the first one we prove two key inequalities, and in the subsequent ones we rewrite 
them in a form more similar to (ITOl i. The subsequent result is contained implicitly in several recent 
papers providing local results about Gauss-Newton method, see e.g. [29, 31J. 

Proposition 1. Let F : D.^Y be a Gateaux differentiable operator. Then: 

(i) if F' satisfies the radius Lipschitz condition of center with L average onU ^Q. (with L 
and U as in Definition\J}, then for all x^U 



(ii) if F' satisfies the center Lipschitz condition with L average at x^ on U C H (with L and U 
as in Definition |2]), then for all x 



Proof. Let x G U and define (p : [0,1] ^ Y by setting (p{t) = F{x^ + t{x — x^)). Clearly is 
differentiable and <p'{t) = F'{x^ +t{x — x^)){x — x^). Then 



F{x^) —F{x) —F'{x){x^ —^)\\ < 2 ll'^* "■^ 



(10) 




(11) 




(12) 




Therefore 
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and hence 

||F(x,)-F(x)-F'(x)(x,-x)|| < / ||F'(x,+f(x-x,))-F'(x)||||x-x,||df. 

JO 

Let us first prove (/). By we get 

/■I r\\^-^*\\ 

/ ||F (x^, +f(x — X*)) — F (x)||||x — x*||df < ||x — x^,|| / / L{u)dudt. (13) 

JO JoJt\\x-x,\\ 

Note that in general, setting r(M) = /Q"L(v)dv, it follows 

[if L{u)du \ pdt = I ( [ L{u)du— [ L{u)du \ pdt 
JO \Jtp J JO V"'o JO / 

= prip)- f'ritp)pdt 

JO 

rP 

= [ur{u)]?. — / r{u)du 

Jo 

"P 

uV{u)du 

P 

L{u)udu (14) 

where we used the change of variables u = tp and an integration by parts. Writing the equality 
obtained in (fT4l) for p = ||x — x* || , (fT3] ) becomes 

||F(x*) — F(x) — F (x)(x* — x)|| < / L{u)udu, 

JO 

so that (/) is proved. 

To show that (//) holds, observe that the center Lipschitz condition ^ implies 

\\F'{x,+t{x-x,))-F'{x)\\ < ||F'(x,+f(x-x,))-F'(x,)|| + ||F'(x,)-F'(x)|| 

rt\\x-x4 f\\x-x4 

< / L{u)du+ / L{u)du 

JO JO 

Reasoning as in the previous case and using the same notations it follows 

||F(x,)-F(x)-F'(x)[x*-x]|| < f\{tp)pdt + r{p)p 

JO 

p 

r{u)du+r{p)p 

p 





rP 

J r{u)du+[{2p-u)r{u)] 



(2p — u)T'{u)du 

\\x-x4 



r\\x-Xt\\ 

■ / (2||x — X*|| — M)L(M)dM. 

JO 



□ 

The following Propositions are a direct consequence of Lemma 2.2 in |,30il . and have already 
been stated in a slightly different form in | )3ll - We include the proofs for the sake of completeness. 

Proposition 2. Given L : [0,/?) — )• M a continuous, positive and increasing function, the function 
Jx defined by setting 



7a:[0,/?)^M, 7a W 



^ f u^L{u)Au ifre{0,R) 

'^^ Jq 

(15) 



is well-defined, continuous, positive and increasing for all A > 0. Moreover yx is constant and 
equal to L/{\+X) if Lis constant, and it is strictly increasing ifL is strictly increasing. Finally 
the following inequality holds for every r € [0,/?) 

{l+X)yx{r)<L{r). (16) 

Proof. Clearly yx is differentiable on (0,/?) since it is a product of differentiable functions and by 
definition 

r^+^y^{r)= f u^L{u)Au. (17) 



Differentiating both members of ([TT] ). it follows 

{l+X)r^yx{r) + ?+^i^{r)=r^L{r), 

therefore 

ri^{r)=L{r)-{l+X)yx{r), (18) 

Thus, if we prove ([T6l ) we also get that yx (r) is increasing. To this aim, taking into account that L 
is increasing, we have 

r^+^y^{r)= f'u^L{u)du< f u^L{r)du = -^^^L{r) (19) 
Jo Jo 1 + A 

from which ([T6l ) follows. Note that if L is strictly increasing the inequality in ([T9] l is strict, there- 
fore in this case, recalling (ITSl ). Yx{r) > on {0,R). On the other hand, if L is constant the 
inequality in ([T9l ) is indeed an equality and Yxir) =0 by (fTSl ) implying that yx is constant on 
(0,/?). The continuity of yx at follows by L'Hospital's rule. In fact, using that L is continuous at 
0: 

^ ^ J^u^L{u)du r^L{r) L(0) 
hm yx (r) = lim ^ — — = lim = 

r^o'^^ ^ r^O rl+^ r^0(l+A)r^ 1+A 

□ 

Using the function 70 introduced in Proposition |2]the center Lipschitz condition with L average 
can be written in the following form, resembling the classical definition of Lipschitz continuity 

||/(x)-/(x,)|| < 7o(||x-x,||)||x-x*||. 
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Proposition 3. Under the assumptions of Proposition^ the function 

( 1 



f ■ [0,R) 



fir) 



r^ Jo 



{lr-u)L{u)Au ifr£(0,R) 



(20) 



3L(0) 



ifr = 0, 



is well-defined, continuous, positive and increasing. 
Proof. The definition of f immediately implies 

r^7^(r) = / {2r — u)L{u)du = 2r L{u)du— / uL{u)du 
Jq Jq Jo 

and differentiating both members of (|2TI ) 

2rf )'(r) =2 f L{u)Au + rL{r) 

Jq 

Dividing by r, and using the notations of Proposition |2j we obtain 

r(fy(r)=2(7o(r)-f(r))+L(r). 
Therefore, in order to prove that / is increasing we just need to show that 

2f(r)<27o(r)+L(r). 
In fact, using the definitions of the functions f and 70 and the monotonicity of L we have: 

rr rr 

2r^f{r)=2 rL{u)du + 2 {r-u)L{u)du 
Jo Jo 

< 2r^7o(r) + 2L(r) / {r-u)du 
Jo 

= 2r^Yo{r) + t^L{r), 
that clearly implies (l22l ). The continuity of f at can be deduced as follows 



(21) 



(22) 



lim f (r) = lim27o(r) - 71 (r) = 2L(0) - ^ = |l(0). 

r^O r^Q 1 1 

relying on the continuity of 70 and 71 proved in Proposition |2] 



□ 



Remark 2. Using the functions 70,71 and / introduced in (ITST l and in (|20] |. the inequality © 
written for F' becomes 



<7o(||x-x. 

The inequalities (ITTI ) and (fT2l ) can be written respectively as 



(23) 



and 

\\F{x,)-F{x)-F'{x){x, -x)\\ < f{\\x-x4)\\x-x,f, (24) 

that generahze the inequahty (ITOl ). 

Note moreover that the functions ?'7o(r), r^/i (r) and r^7^(r) are always strictly increasing and 
if L is a constant function equal to L, then 

L 3 
7o(r)=L, 7iW = 2' /('') = 2^- 

Remark 3. It is worth noting that, even though only a Gateaux differentiability has been required, 
the previous inequalities together with the hypothesis on the function L implies Frechet differen- 
tiability at x^. 

3.2 Generalized inverses 

In this section we collect some well-known results regarding the Moore-Penrose generalized in- 
verse (also known as pseudoinverse) of a linear operator A. They will be useful in the rest of 
the paper. For the definition and a comprehensive analysis of the properties of the Moore-Penrose 
inverse we refer the reader to IITtI . 

Assume that A e L(X,y) has a closed range. The pseudoinverse of A is the linear- operator 

G L(F,X) defined by means of the four "Moore-Penrose equations" 

AAU=A, A^AA'^=A\ {AA'^)*=AA\ (aU)*=AU. (25) 

Denoting by P]m(a) ^rid Pr(a) the orthogonal projectors onto the kernel and the range of A respec- 
tively, from the definition it is clear that 

aU = /-P;v(a), AA'^=PR(Ay (26) 

In case A is injective, N{A) = {0} and A^A = /, that is A^ is a left inverse of A. Furthermore, for 
each A G L(X,y) the following statements are equivalent: 

• A is injective and the range of A is closed; 

• A* A is invertible in L(X,X). 

and if one of those equivalent conditions is true then A''" = (A*A)"^A* and HA^p = ||(A*A)"^||. 
The following lemma gives a perturbation bound for the Moore-Penrose pseudoinverse, see 



Lemma 1. Let A,B G h{X,Y) with A injective and R{A) closed. If \\{B — A)A^\\ < 1, then B is 
injective, R{B) is closed and 

IIB'IK «^'« 



l-||(B-A)At|r 
Moreover 

IIB^-A^II < \/2||A"''||||Bt||||B-A||, 

and therefore 

B^ -A^ < V2 " „'' „ ^. 

" 1- At LB-A 
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3.3 The proximity operator 

This section consists of an introduction on proximity operators, which were first introduced by 
Moreau in |[33l . and further investigated in ||34] 1351 as a generalization of the notion of convex 
projection operator. Let H : X ^ X be a continuous, positive and selfadjoint operator, bounded 
from below, and therefore invertible. Then we can define a new scalar product on X by setting 
{x,z)h = {x,Hz)- The corresponding induced norm \\-\\h is equivalent to the given norm on X, 
since the following inequalities hold true 

— ^Il-xf < < ll^llll-xf • (27) 



\\H 

The Moreau-Yosida approximation of a convex and lower semicontinuous function <p : X — )■ 
M U {+°°} with respect to the scalar product induced by H is the function M(p : X — )• M defined by 
setting 

M^{z) = M\j{x) + \\\x-z\\iy (28) 

For every z&X, the infimum in equation ( |28l ) is attained at a unique point, denoted prox^(z). 
In this way, an operator 

prox^ : X — ^ X 

is defined, which is called the proximity operator associated to tp w.r.t. H. In case H = I is the 
identity, the proximity operator is denoted simply by prox^. Writing the first order optimality 
conditions for (|28] |. we get 



p = prox^(z) ^Oed(p{p)+H{p-z)^Hze{d(p + H){p), (29) 

which gives 

prox'^{z) = {H + d(p)-\Hz). 

We remark that the map {H + d(f))^^ (in principle multi-valued) is single- valued, since we 
know that the minimum is attained at a unique point. 

Lemma 2. The proximity operator prox^ :X ^ X is Lipschitz with constant y/\\H\\ \\H^^ \\ with 
respect to ||-||, namely 



||prox^(zi)-prox^(z2)|| < ^||//||||//-i||||zi -Z2II. (30) 

Proof. Being the proximity operator firmly nonexpansive with respect to the scalar product in- 
duced by H (see e.g. Lemma 2.4 in [1 IJ) we have 

||prox^(zi)-prox^(z2)||ff < ||zi -Zi\\h- 
Using the inequalities in ( |27l ) relating ||-|| and ||-||// we get the desired result. □ 



It is also possible to show that in some cases the computation of the proximity operator with 
respect to the scalar product induced by H can be brought back to the computation of the proximity 
operator with respect to the original norm. In particular, the following proposition holds. 
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Proposition 4. Let A G L(X, y). Let us suppose A to be injective with closed range. Set H = A* A 
and assume 9 : X — )■ MU {+00} a proper, convex and lower semicontinuous functional. Then 

prox^ =A^prox^„^iA 

Proof. Being A injective, H is positive and invertible, thus 

prox^ -.X^X prox^(;c) = argmin + ^||z -x||^| 

where 

\\z-x\\]j = {A*A{z-x),z-x) = {A{z-x),A{z-x)) = ||A(z-x)f . 

Therefore 

prox^(;c) = argmin{(p(z) + -||A(z-;c)f}. (31) 

On the other hand, since (p oA^ : F — 5- MU {+°°} is convex and lower semicontinuous the corre- 
sponding proximity operator with respect to ||-|| is well-defined and 

prox^„^i -.Y prox^„^t(3') = argmin |(p(Atf) + -yf | 

If we set X = prox^(;t), by (|3TI ) we obtain 

<p(l) + ^||Al-Axf < (p(z) + ^||Az-Axf VzGX. 

Moreover, by setting y = Ax, taking t G Y, with t = ti+t2 such that t\ G R{A) and t2 G R{A)^ and 
z = A^f G X, we have 

((>{A^t) + ^ik -^W^ = <p(^^0 + ^Iki +^2 -Ax\\^ 

= (p{A^t)+'^\\PR^J,^t-Ax\\'^ + \\t2f 

> (p{A'^t) + ^\\AA'^t-Axf 
= (p{z) + ^\\Az-Axf 

> <jp(x) + ^||Ax-Ax||^ 

= (p{A'^y) + ^\\y-Ax\\\ 



We finally get 
and thus using 



y = ari 

teY 



gmin |(i()(A+0 + ^lk-Axf| = prox^„^i (Ax) 
prox^ (x) = X = A'^Ax = A^y = A^prox^^^t (Ax) . 



□ 



11 



Since in the sequel the proximity operators will be computed with respect to a variable norm 
II -11//, we are interested in the behavior of the proximity operator when H varies. 

Lemma 3. Let Hi and H2 two continuous positive selfadjoint operators on X, both bounded from 
below. It holds 

llprox^'(z) -prox^H^)!! < \\H^'\\\\{Hi-H2){z-WO^'^-{z))\\. (32) 
Proof. By (|29l) it follows 

Hi{z-^mx^{z)) G 5(p(prox^'(z)), / = 1,2. 
Then, by definition of subdifferential the following inequalities hold true 

<P(prox^'(z)) >'P(prox^'(z)) 

+ (//i (z - prox^' (z)),prox^2 _ prox^i (2)) 

(p(prox^'(^))>'P(prox^'(z)) 

+ (//2(z-proxJ^(z)),proxJ(z) -proxjHz)). 

Summing up them, we obtain 

> (//2(z-prox^^(z)) -//i(z-prox^'(z)),prox^'(z) -pro<Hz)), 
and equivalently 

(//iprox^' (z) -//2prox^nz),prox^' (z) -pro<'(z)) < 

{{Hi - //2)z,prox^' (z) - prox^2 (z)). 
Adding and subtracting the same term, the previous inequality can also be written as 

(//i(proxJ'(z) -proxjHz)),proxJ'(^) -prox^Hz)) < 

( [Hi - H2) (z - proxj^ (z) , prox«> (z) - prox^^ (z)) , 

from which (l32l) follows. □ 



Note that in the previous lemma Hi and H2 play a symmetric role, so that they can be inter- 
changed. 

Remark 4. Combining (l30l ) and (l32l ). we get: 

llprox^'zi -prox^2z2|| < ||prox^'zi -prox^'z2|| + ||prox^'z2 -prox^2z2|| 

<(||//i||||//ri)^/'||zi-Z2|| (33) 
+ \\H;'\\\\{Hi- H2){z2 - proxf Z2) II , 

for every zi,Z2 G ^ and Hi,H2 continuous and positive selfadjoint operators on X, bounded from 
below. 
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4 Setting the minimization problem 



In this section we collect some basic properties of the solutions of problem {^). The following 
will be standing hypotheses throughout the paper. 



(SH) 



F : n C X — F is Gateaux differentiable 

/ : X — )■ MU {+00} is proper, lower semicontinuous and convex. 



Recall that J proper means the effective domain dom(7) := {x G X : J{x) < +00} is nonempty. 

Without loss of generality, we shall assume y = in the problem (^), since the general case 
can be recovered just by replacing F with F — y. Thus, hereafter, the following optimization 
problem will be considered 

min^||f(x)f + 7(x):=<I>o(x). (^0) 

The functional is in general nonconvex and searching for global minimizers turns out to be a 
challenging task. Therefore, the focus of this paper is on local minimizers of <I>o, whose existence 
shall be assumed from now on. Generally speaking, Gauss-Newton methods are of a local charac- 
ter, and allow to find a local minimizer. As it is well-known a local minimizer x.^, of is a point 
such that x-f e dom(7) and there exists a neighborhood U of x,, such that <I>o(x,,) < <I>o(x) for 
all xeU. 

By the way, hypotheses (SH) are not enough to guarantee the existence of a global mini- 
mizer of the problem (^o)- Such existence can be proved relying on the Weierstrass theorem 
as soon as we impose F to be weak to weak continuous and <I>o (weakly) coercive, namely 
lim||_,||^+„<I>o(x) = +00. 

We start by providing first order conditions for local minimizers. 

Proposition 5. Suppose (SH) are satisfied and let x^ £ D. be a local minimizer of^Q. Then the 
following stationary condition holds 

—F'{x^)*F{x^) G dJ{x^). 
Moreover, ifF'{x^) is injective and R{F' [x^)) is closed, then x.^, satisfies the fixed point equation 

x^ = prox^*-'^* •* (x,, — F' {x^)^ F {x^)) , 

withH{x^) :=F'(x,)*F'(x,). 

Proof. Suppose that x^ is a local minimizer of Oq. Denoting by (x* , v) the directional derivative 
of at X* in the direction v G X, which exists thanks to {SH), the first order optimality conditions 
for X* impUes 

<J>o(^*,v)>0 VvEX. (34) 
As a consequence of the differentiability of F and the convexity of / ( |34l ) can be rewritten as 

-F'(x*)*F(x,)v</(x,v) VveX, 
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and consequently, by Proposition 3.1.6 in [9], also as 

-F'{x^yF{x^) G dJ{x^), (35) 

which is the stationary condition of the thesis. To prove that satisfies the fixed point equation 
note that adding H{x^)x^ to both members of ( [35] ) we have 

//(x,,)x* — F'(x^,)*F(x*) G (//(x*) + 57)(x*). 

Since H{x^) is invertible, then the previous equation can be also rewritten as 

//(x,)(x,-//(x,)-1f'(x,)*F(x,)) G (//(x,) + 5/)(x,). 



RecaUing equation ( |29l ) and the properties enjoyed by the pseudoinverse we obtain the second 
assertion. □ 

5 The algorithm - convergence analysis 

In this section we state the main result of the paper, consisting in the study of the convergence of 
a generalized Gauss-Newton method for solving problem (^o)- The flavor is similar to the most 
recent results concerning the standard Gauss-Newton method, proved in 13T1 . We start describing 
some basic properties of the proposed algorithmic framework. 
Fix xq G dom(7), and then, given x„, define x„+i by setting 

1 2 
x„+i = argmin-||F(x„) + F (x„)(x-x„)|| +7(x). (36) 
xex ^ 

Note that since the quantity inside the norm has been linearized, this problem can be solved explic- 
itly, for instance using first order methods for the minimization of nonsmooth convex functions, 
such as bundle methods or forward-backward methods (see lITQlfTTl ). Writing down the first order 
optimality conditions we will get a similar formula to the one in Proposition [5]for a minimizer x*. 

Proposition 6. Suppose F'{xn) is injective with closed range and set H{x„) = /^'(x„)*/^'(x„). 
Then, the formula (1361) defining x„+i is equivalent to 

Xn+ 1 = proxf ^ (x,j -F'{Xn)^F{Xn)). (37) 

Proof. Thanks to the assumptions made on F'{x„) the operator H{xn) is invertible. Writing the 
first order necessary conditions, which are satisfied by x„+i we obtain 

G F' {Xn)*[F{Xn) +F'{Xn){Xn+\ - X„)] + 57(x„+i ) 
F'{Xn)*F'{Xn)Xn-F'{Xn)*F{Xn) G (F'(x„)*F'(x„) + 57) (x„+i ) 
^ X„+i = {F'{Xn)*F'{x„)+dJ)-\F'{x,;)*F'{x,;)Xn-F'{Xn)*F{Xn)) 
<S=^> X„+i = {F'{Xn)*F'{Xn) + djy^F'{Xn)*F'{Xn) (x„ - F'(x„)"^f (x„)) 

<^ x„+i = prox^'"'"^ (x„ - F'(x„)^F(x„)) 

□ 
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In the next theorem we provide a local convergence analysis of the proximal Gauss-Newton 
method, under the generalized Lipschitz conditions on F' introduced in Section 13.11 The proof is 
postponed to Section [6l 

Theorem 1. Suppose that (SH) are satisfied. Let U (^D.be an open starshaped set with respect to 
Xif, where £ dom(7) DU is a local minimizer of(£>o. Moreover assume 

1. F'{x^) is injective with closed range; 

2. F' : D. X ^ L,{X ,Y) is center Lipschitz. continuous of center x^ with L average onU (L as 
in the definition\2\and increasing); 

3. [{l + V2)K+l]al3^L{0) < \, where a = \\F{x^)\\,p = \\F'{x^y\\, k = \\F' {x^Y \\\\F' {x^)\\, 
the conditioning number ofF'{Xi,). 

Define R and q : [0,^) — )• M+ by setting R = sup{r G (0,/?) : 7o(r)r < l/j8} and 
^^^^ ^ j3 f Mo{r)f{r)r^ + Kf\r)r ^ (1 + ^/2)a^^yo{rfr 



l-jS/oW'-l (l-i87o(r)r) l-jS/oWr 

^ [(l + ^/2)>c+l]«/3yo(r) 
l-j87o(r)r 

The function q is continuous and strictly increasing. If we define 

f = sup{r e (0,^] : q{r) < 1}, (38) 
and wefixrG M, with <r <r, such that Br{x^) ^ U, we get that the sequence 

Xq G Br{x^), 

Xn+i = proxf (x„ - F'(x„)"^F(x„)) 

with H{xn) '.= F'{x„)*F'{xn), is well-defined, i.e. x„ G Br{x^) and F'{x„) is injective with closed 
range and it holds 

1 1 -^n 1 1 ^ ^0 1 1 1 1 ' 

where qo := ^(||x — xo||) < 1. 

More precisely, the following inequality is true 

ll-^n+l II — ^2 \\-^n II ~i~ ^1 ||-^rt II ) 



for constants Ci > and C2 > defined as 

C, : 



C2 



[(l + ^/2)>c+l]aj3^yo(p,J _ 
^PfiPxo) + (1 + V2)aP'roiPxJ^ + P^7o{Pxo)fiPxo)Px, 



(l-j87o(pA-o)pAo)2 
with = ||x — xo||. 
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Since r is chosen as the biggest value ensuring q{r) < 1 (a sufficient condition making the 
Gauss-Newton sequence convergent), r can be thought as the radius of the basin of attraction 
around the local minimum point x,,, even though in general we can't prove the optimality of this 
value. 

Remark 5. An analogous theorem is true if in the assumption 2 we suppose F' to satisfy the 
radius Lipschitz condition. All the statements remain true, just replacing / with 71. For instance 
the expression of q{r) becomes 

. P [ i8yo(r)yi(r)r^ + >cyi(r)r (l + ^/2)«j3^yo(r)V 

^^'■^ l-Pro{r)r\ l-Pro{r)r + I - Pro{r)r 

^ [(l + V2)>c+l]«/3yo(r) | 
l-j8yo(r)r /■ 

Remark 6. The hypotheses we impose are in line with the state-of-art literature about classical 
Gauss-Newton method (/ = 0), see fSP]. It is worth noting that the expression of r is not affected 
by the choice of J. On the other hand, the presence of the function / reduces the radius of con- 
vergence of the Gauss-Newton method. Indeed, the expression for r obtained in (1381 ). which is 
valid also in the case 7 = is always smaller than the maximum radius of convergence that can be 
derived from equation (3.4) in [31 1, namely 

ro = sup{r G (0,^) : qo{r) < 1}, 

with 

The reason is that the bound (|30] | we use, is not sharp in case / = 0, and this causes an additional 
term in the expression of q{r). 

Conditions ensuring quadratic convergence. As in the classical case, also with the additional 
term J, for zero residual problems quadratic convergence holds. In fact, from the expression of 
Ci, we see that Ci = if a = 0, i.e. F{x^) = 0. 



5.1 The case of constant average L 

In case the function L is constant, we can derive also an explicit expression for the maximum ray 
of convergence r. 

Corollary 1. Let the assumptions of Theorem\l}be satisfied and moreover assume F'{x^) to be 
center Lipschitz continuous of center with constant average L on U. Define q : [0, 1/ (j3L)) — t- 
M+ as 

P f HpL^r^ + KLr) (1 + V2)«j3^L^r [(1 + V2)>c+ l]«j3L 
l-j8Lr\ 2(l-j3Lr) ^ l-pLr ^ I - pLr 
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which is continuous and strictly increasing in its domain. If we define 
h=[{\ + V2)K+\]a^'^L (<1), 
1 



2 + ^ + (1 + V2)aj82L^ + J + ^ + (1 + ^f2)a^^L^ + 2(1 - h) 



and we fix r G M with < r <r such that Br{x^) C JJ the conclusions of Theorem\l\hold with 

[(l + V2)jc+l]a/32L_ 



C2 = P 



3k+{\ + V2)al5^L^ + dpL^p,, 



2(1 -^Lp,J2 

Proof. Using the expressions of jo and found in Remark|2l one can easily show that R = 1/ (j8L) 
and q can be written as in ( |39] ). 

As before ^ is continuous and strictly increasing on the interval [0, 1 / (jSL)), and 

q{0)=h<\, lim ^(r)=+oo. 

Therefore r defined in (1381 ) is the unique solution in (0, 1 / (j8L)) of the equation q{r) = 1. We are 
going to find that point explicitly by solving the equation ^(r) = 1. The latter is equivalent to the 
following quadratic equation 

+ (4 + 3JC + 2(1 + V2)al5^L)z - 2(1 -h)=0, 

with z G [0, 1), z = ^Lr. That equation has the two distinct solutions 



z = -(^ + ^K+{\ + V2)ap'^]^±^^[2 + ^K + 2[\ + ^/2)a^^l)j +2{l-h). 

Of course, we discard the negative solution, and we keep the one with the plus sign, which can be 
easily checked to belong to (0, 1). 

□ 

Along the same line, a similar result concerning the case of radius Lipschitz continuity can be 
proved. 

Corollary 2. Let the assumptions of Theorem\J\be satisfied and moreover assume F'{x^) to be 
radius Lipschitz continuous of center x^ with constant average L on U. Define q : [0, 1/ (j3L)) — )• 
as 

j8 (pL^r'^ + KLr (\ + V2)al5^L^r [(1 + V2)jc+ l]a/3L 
^v) = — irr. ^ -^-i — irrr + ^ — ^ri + 



I - pLr { 2{l- pLr) \-pLr l-jiLr 
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which is continuous and strictly increasing in its domain. If we define 

h=[{l + ^/2)K+\]ap'^L[< 1), 
1 



2 + 1 + (1 + \/2)a;3^Z.) - J (2 + 1 + ( 1 + v^)a(32z.) ' - 2( 1 - ft) 



and we fix r with < r < r such that Br{x^) C U the conclusions of Theorem\l\hold with 



,>c+(l + V2)«j3^L^ + i8LX 
2(l-j8Lp,„)2 



Ci 
C2 

forr<l/{m- 

6 Proof of Theorem [1] 

We are going to state some auxiliary results for proving convergence of the algorithm discussed 
in the previous section. The following proposition will be one of the building blocks to show that 
convergence holds. 

Proposition 7. Let G : D C X ^ X, be a mapping andx^ G D a fixed point ofG. Let U QD be an 
open starshaped set with respect to G U. Assume G to satisfy the inequality 

||G(x) — G(x*)|| < ^(||x — x,,||)||x — x^ll, forallx&U (40) 

for a given increasing function q : [0,R) — )• [0, +°°), continuous at and such that q{0) < 1. Define 

r = sup{r G (0,^) : q{r) < 1}. 

Then r > and given r G M with < r <r and B,-{x^) C U, it follows G{Br{x^)) Q Br{x^), thus, 
given xo G Br{x^) the sequence defined by setting x„+i =G{x„) is well-defined. Moreover, denoting 
qo = q{\\xo — x*||) it holds qo <i and 

II ^ ^0 ll"^0 II * 

Proof. First note that f = sup{r G (0,/?) : q{r) < 1} > being q{0) < 1 and q continuous at 0. 
Fix r£W,0<r<R such that Br{x^) ^ U and x G Bi-{x^). Then by definition of r, (7(||x — x,, ||) < 1 
and therefore (l40l ) implies that 

||G(x) — x*|| < ||x — x*|| < r, 

i.e. G(x) G Br{x^). Thus G{Br{x^)) ^ Br{x^) and a sequence can be defined in Br{x^) by choosing 
Xo G Br{x^) and setting x„+i = G(x„). Being ||x„ — x* || < r, again from the definition of r we have 
^(||x„ — x*||) < 1, and from (l40l ) we get 

1 1-^/7-|- 1 1 1 — 1 1 ^ (-^/l ) 1 1 ^ ^ ( I \ 11)1 \ -^n 1 1 ^ 1 1"''''^ 1 1 * 
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This implies that^(||;ic„+i — Jc* ||) < — ||), since ^ is increasing. Therefore, denoting q{\\xQ—Xi 
qo we get q{\\xn — x*||) < < 1 for all n G N and 

\\xn+i -x^W < q{\\xn -x*||)||x„ < qo\\xn < . . . < ^oll-^o --x*!!- 

□ 

We now introduce some notations, allowing for rewriting the conditions which have been 
described in Proposition |5]for a local minimizer of <I>(). Define G and G by setting 

G{x)=x-F\x)'^F{x) and G(;c) = prox5^*'^(G(x)), (41) 

where H{x) = F' (x)* F' (x). The domain of G and G is the subset D of 11 defined as 

D = {xeD. : F'{x) is injective and R{F'{x)) is closed}. (42) 

If jc* G D is a local minimizer of (=^o) the fixed point equation of Proposition |5]can be restated 
by saying that x,, is a fixed point for G, namely 

X* = G{x^). (43) 

Proposition 8. Assume (SH) and let x* be a local minimizer of^o belonging to D. Suppose that 
U <ZD. is open and starshaped with respect to x^. Moreover assume 

i) F' . D. Q X ^ L(X,F) is center Lipschitz continuous of center x* with L : [0,R) — t- R+ 
average onU ^ D.; 

ii) a = ||F(x,)||, j8 = \\F'{x^y\\ andK=\\F'{x^y\\\\F'{x^)\\. 
Then, defining R by setting 

R = sup{r£{0,R) : 7o(r)r< 1//3} 
it follows that for all r £M with < r <R andBr{x.f) ^U, G satisfies 

\\G{x) — x*|| < g(||x — x*||)||x — x*||, 
for all X G ^^(x,,), where q : [0,^) — )• M+ is defined as 



P ( l5ro{r)f{ry + Kf{r)r (1 + ^/2)aj32^o( 



l-j87o(r)ri (l-jS7o(r)r) l-jSyoWr 

[{\ + V2)K+l]al5Yo{r) 



+ ■ 



l-j87o(r)r 
and it is continuous and strictly increasing. 
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Proof. Since is a local minimizer of <I>o and x* G D, with D defined as in (1421 ). is a fixed point 
of G (see (|43])). therefore 

G(x,) -proxf^''*^G(x,) = G{x^)-x^=F'(x^)^F{x^). (44) 

Fix r G M, with < r < ^ such that Br{x^) C JJ and take x G ^^(x^). Adopting the notations of 
Proposition m as noted in Remark |2l from the center Lipschitz hypothesis we get 

||F'(x)-F'(x,)||||F'(x,)^|| <7o(||x-x,||)||x-x,||j8. 

Recalling that Remark |2] ensures that the function p i— )• p7o(p) is continuous, strictly increasing 
and takes value in 0, we have that ^ > and 

||F'(x)-F'(x,)||||F'(x,)t||<jS7o(r)r<l, 

and thus applying Lemma[T] F'{x) is injective, with closed range, and 

\\F'{x)'^\\<- „ ^, , , where p., = ||x -x* ||. (45) 

Applying inequality (1331 ) with H\ = H (x), H2 = H (x^), zi = G{x) and Z2 = G{x^), and taking into 
account ((44)) we get 

||G(x) -x*|| = llproxf ")(G(x)) -proxf "*)(G(x,))|| 

<(||//(x)||||//(x)-i||)'/'||G(x)-G(x,)|| 

+ ||//(x)-'||||(//(x) -//(x,))(G(x*) -proxf^-'*^G(x,))|| 

= (||//(x)||||//(x)-l)'/'||G(x)-G(x.)|| 
+ ||//(x)-'||||(//(x)-//(x,))F'(x,)tF(x,)||. (46) 

Moreover 

||//(x)|| = ||F'(xrF'(x)|| = ||F'(x)f 
||//(x)-i|| = ||[F'(x)*F'(xri|| = ||F'(x)t||2. 

Recalling the properties of the Moore-Penrose generalized inverse in (1261 ) and Lemma[T]we get 

{H{x) -Hix,))F'{x,y = (F'(x)*F'(x) -F'(x,)*F'(x,))f '(x,)t 

= F'{x)*F'{x)F'{x,y -F'{x,)*F'{x,)F'{x,y 
= F'{x)*F'{x)F'{x^)^ -F'{xyPii(^F,(^^_^)) 

+ F'{^)*Pr{f'{x,)) - F' {x*)*Pr{f'(x,)) 
= F'{xr{F'{x)-F'{x,))F'{x,y 

+ (F'(x)-F'(x,))*/'«(^,(,,)), 
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therefore, 

\\iH{x)-H{x,))F'ix,y\\ < (||F'(x)||||F'(x,)^|| + l)||F'(x)-F'(x,)||. (47) 
Hence, substituting in (l46l ) the bound derived in (|47] ) we obtain 

llGW-x.ll < ||F'(x)||||/^'(x)t||||G(x)-G(x,)|| (48) 

+ \\F'{xYf {\\F'{x)\\\\F'{x.,Y\\ + 1) \\F'{x)-F'{x.M\F{x,)\\. 

On the other hand, thanks to the properties of the Moore-Penrose pseudoinverse reported in (126^ 
and the injectivity of F'(x) 

G{x) — G{x^) =x — x^ —F'{x)'^F{x) + F' {x^)^ F {x^) 
= F'{x)\f'{x){x-x^)-F{x)+F{x^)] 
+ {F'{x,)^-F'{x)^)F{x,) 

and thus, using Lemnia[I] 

\\G{x)-G{x^)\\ < \\F'{xY\\\\Fix,)-F{x)-F'ix){x,-x)\\ 

+ ||F'(x,)1--F'(x)^||||F(x,)|| 
< ||F'(x)"^||||F(x,)-F(x)-F'(x)(x, -x)|| 

+ V2\\F'{xf\\\\F'{x,)'^\\\\F'{x)-F'{x,)\\\\F{x,)\\ 
= \\F'{xY\\{\\F{x,)-Fix)-F'{x){x,-x)\\ 

+ V2||F'(x,)-f||||F(x.)||||F'(x)-F'(x,)||} (49) 

Substituting (|49ll in (1481 ) 

\\G{x)-x4<\\F\xYf{\\F'{x)\\{\\F{x,)-F{x)-F'{x){x,-x)\\ 

+ (l + V2)||F'(x,)t||||F(x,)||||F'(x)-F'(x,)||) (50) 
+ ||F'(x)-F'(x,)||||F(x.)||]} 

Taking into account Remark [2l we can rewrite inequahty (l50] l as 

||G(x) — x*|| 

<||F'(x)tf {||F'(x)|| [fip,)p^ + {l + V2)\\F\x,y\\\\Fix,)UMp. 
+ ||F(x,)||7o(p,-)p,} 

To find a bound for the quantity ||F'(x)||, recall that k is the conditioning number of F'(x^,), 
i.e. K := ||F'(x,,)^|| ||F'(x*)||, and by the triangular inequality and Remark dU we get ||F'(x)|| < 
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\F'{x)-F'{x^)\\ + \\F'{x^)\\ < YoiPx)Px + K"/j8. Thus, recalling (|45}, we finally obtain 

||G(x) — x*|| 

fi2 , |- 

^ T. — o r X n2 (ro(Px)p.v + fc/jS) f{p,)p^ + (1 + \/2)i3a7o(p,)p, 
(l-i87o(p;.)Px) L 

a7o(p;.)pA}, 



+ 1 



or, equivalently 

\\G{x) — < 



l-i37o(px)p.v 



j3yo(px)f (P.v)px + ^fiPx)px 

l-jS 70 (Px)p.v 



, (l + ^/2)«i8^7o(p,)^P:, , [{\ + V2)K+\]al5Yo{px) \ , 

+ . n / N + . n / N ^ Pa- 



l-i87o(PA)Px 



l-i87o(P.v)Px 



1 1 11/11"''' 1 1 5 



where we set 



l-PYo{r)r{ (l-i87o(r)r) 



+ 



+ 



l-l5Yo{r)r 
[{\ + V2)K+l]apYo{r) 



l-l5Yo{r)r 

Finally, it is easy to prove that q is continuous and strictly increasing relying on Remark |2] 



□ 



Proof of Theorem \J} 

Let G and G be defined as in (|4TI) and define R and q as in Proposition [8] Now fix r < r such 
that ^ f^- Then, thanks to hypothesis 3), it is possible to apply Proposition Hand to get the 

first part of the thesis. Finally, relying on the structure of the function q shown in Proposition [H 
and denoting p^^ = — X()||, the expression of the constants Ci,C2 easily follows. 

7 Applications 

Algorithm (|37] ) is a two-steps algorithm, consisting of the classical Gauss-Newton step followed 
by a "7-projection" in a variable metric. In this section the general framework shall be specialized 
to solve constrained nonlinear systems of equations in the least squares sense. We remark that due 
to hypotheses of Theorem [T] we are dealing with regular problems in the sense of Bakushinskii 
and Kokurin Q. In the finite dimensional case this implies the number of equations to be greater 
than the number of unknowns. This subject has been studied in ||6l|24j|22j|43l (see also references 
therein). Denoting by C a closed and convex subset of X, we consider the problem 

mm\\F{x)f, (51) 
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which can be cast in our framework by setting J{x) = lc{x), where Iq denotes the indicator function 
of the set C, i.e. lc{x) = for x G C and lc{x) = +00 otherwise. The proximity operator of ic with 
respect to H{x„) = F'(x„)*F'(x„), turns out to be the projection onto C w.r.t. the metric defined by 
H{xn), and therefore algorithm <Bl] reads as follows 

Xn+l =Pc^'"\xn-F'{XnfF{x„)). (52) 

Since in general a closed form of the projection operator is not available, a further algorithm 
is needed for solving the projection task, which adds an inner iteration to the main procedure. 
We choose the forward-backward algorithm 111]. Though there are many other methods for that 
purpose, we do not carry out any comparison among them, because this is beyond the scope of the 
present paper. By definition of proximity operator, given H = A* A, A injective with closed range, 
we have 

P^(z)=prox(^.(z) 

= argmin|ic(v) + ^||v-z||^| 

= argmin I ic (v) + ^ 1 1 Av - Az f I . 

If Pc denotes the projection onto the convex set C, now with respect to the original metric of the 
space X, the sequence defined by 

voGX 

Vk+i = Pcivk-(yH{vk-z)), 
with a < 2/||A|p, is strongly convergent to the point Pq{z) ll39l . Eventually, the full algorithm is 
"xo G C 

Zn — P ip^n ) P {^n ) 

rvo,„GC, o„<\/2\\P'{x,)f (53) 

[vk+l,n = Pc{Vk.n - OnP'{Xn)*P'{x„){vk^„-Zn)) 

X„+i = limVA:.„. 
L k 

It is worth noting that the inner iteration is not required when Zn belongs to C. Indeed, in that case, 
the projection leaves Zn untouched and the full step of the algorithm (l52l ) reduces to the classical 
Gauss-Newton step. Such situation asymptotically occurs when x* is internal to C. 

Algorithm (1531 ) requires explicit evaluation of the projection Pc, which can be done for simple 
sets, like spheres, boxes, etc. Particularly relevant from the point of view of the applications — see 
||6l and references therein — is the case of box constraints in M". We point out that when Pc can 
be computed explicitly the algorithm generates a sequence of feasible points, no matter when the 
inner iteration is stopped. This feature can be useful in forcing the sequence of iterates to remain 
in regions where the function is well-behaved, avoiding the Gauss-Newton step to lead to sites 
where the derivative is ill-conditioned. 
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8 Numerical experiments 



This section summarizes the results of the numerical experiments we carried out in order to verify 
the effectiveness of algorithm (1531 ) for solving real-life constrained nonlinear least-squares prob- 
lems. In particular, we consider the case of box constraints, namely 

min — ylF, a<x<b, 

xeR" II ^ ^ ''II ' - - ' 

where a and b are in W,a<b and C = U'Li [ai,bi], F :a^C ^R"". 

The aim of the tests is to illustrate the behavior of our algorithm on some representative exam- 
ples and show that it can be successfully applied to real problems. The algorithm is implemented 
in MatLab, and the convergence tests 

are used with precision £ = 10^^^. 

We remark that the implementation of the algorithm (l53T l computes the projection only ap- 
proximately, meaning that the internal iteration is stopped either because the required precision 
has been attained or because an a priori fixed maximum number of iterations has been reached. 
For this reason, the projection step depends on the algorithm selected to that aim and the forward- 
backward algorithm is just one choice among several possibilities. Furthermore, the number of 
evaluations of F and F' depends only on the number of outer iterations. Therefore we provide just 
the number of outer iterations needed to reach the target precision as a measure of our method's 
performance. Yet, the number of inner iterations does affect the number of outer iterations. In 
fact, in our experiments we observed that, even though the algorithm is quite robust with respect 
to errors in the computation of the projection, the number of outer iterations can increase if the 
required inner precision is not attained (inner iterations reach the maximum allowed). 

The experiments are performed on some standard small residual test problems. One group of 
them is taken from [ 15 1 and a second group comes from the extensive library NLE [41], which is 
accessible through the web site: www. polymath-software . com/library. We considered only 
problems for which the solution (or a good estimate of it) is known in advance and for which 
the Gauss-Newton method is known to be effective — since our proposal in fact extends the 
classical one. The problems we select in the first group are Rosenbrock, Osbornel, 0sborne2 
ll32l and Kowalik [23 |. They are actually unconstrained problems, to which we added some box 
constraints set up in order to make the provided solution fall on the boundary of the box (faces, 
edges, vertices, etc.). Besides, on Rosenbrock's example we tried out our method also in case the 
global minimizer is kept outside the fixed box. 

The remaining problems, Twoeq6 and Teneqlb, come as truly constrained and are labeled as 
"higher difficulty level" in the NLE library. Unlike the first group, here the constrained minimizers 
of Twoeq6 and Teneqlb lie in the interior of the feasible set. Observe in addition that the given 
constraints define a convex set that is not closed. More specifically, in Teneqlb example, the 
feasible region is the positive orthant of excluding four coordinate hyperplanes where the 
first derivative is undefined. We overcome this difficulty by shrinking slightly the feasible region 
of a small amount 5 

Cg = \{xi,. . . ,xio) ■ Xi> 5 for / = 1, . . . ,4, X,- > for / > 5 >, 
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and solving the problem in the closed and convex set C§ with 5 = 0.0001. The same trick has 
been used also for solving problem Twoeq6 too. 

The experimental protocol is different for the two test groups: in the former group, 20 points 
belonging to the box are randomly chosen as initial guesses and the average number of required 
iterations is provided; whereas in the latter one the algorithm is fed with two critical initializations 
reported in the NLE's problem description. The results are collected in Table[T] 

In all tests the algorithm reached the solution up to the required precision, and we did not detect 
any significant variation in the number of iterations depending on the starting points. Along the 
path we checked the condition number of F', observing that it always keeps bounded from above, 
the problem thus remaining well-conditioned. In case of 0sborne2, we saw that the classical 
Gauss-Newton method does not converge for some initializations, due to ill-conditioning of the 
derivative F'. We were able to correct this behavior by setting the constraints properly around the 
known minimizer. 

9 Conclusions 

This paper shows that the local theory on the convergence of the Gauss-Newton method can be 
extended to deal with the more general case of least squares problems with a convex penalty. 
The main theoretical result demonstrates that, under weak Lipschitz conditions on the derivative, 
convergence rates analogous to those existing for the standard case can be derived. An explicit 
formula for the radius of the convergence ball is also provided. A valuable application we propose 
concerns nonlinear equations with constraints. Our algorithm has been found effective and robust 
in solving such problems as shown in several numerical tests. Both the cases of solutions on the 
boundary of the feasible set as well as solutions in its interior as been treated successfully. 
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Table 1 : Results of numerical experiments on the test problems specified in the first column for 
the box constraints given by a and b with starting point xq. The point x^, is the detected minimizer, 
which we show with five digits precision for conciseness. 

Function I /; I m I a I h I a'o | a's | avg. n. iter. 



Rosenbrock 



Kowalik 



Osborne 1 



0sborne2 



Twoeq6 



Teneqlb 



11 



10 



11 



31 



65 



10 



-3 
-2 

0.1928' 
0.1916 
0.1234 
0.1362 



0.3754' 
1 

-2 
0.01287 




1.31 






1.4 


0.4314 






0.8 


0.6336 






1 


0.5 






1 


0.5 






1 


0.6 






3 


1 






5 


4 






7 


2 






2.5 


4.5689 






5 


5 






6 



0.0001 
0.0001 

"0.0001 
0.0001 
0.0001 
0.0001 















■ 3 ■ 
0.8 

1 
1 
1 

1' 

2 


1 
1 



0.9999 

-|-oo 

-[-oo 
-|-oo 
-|-oo 

-j-OO 
-j-OO 

-(-oo 
-[-oo 

-j-OO 

-|-oo 
-|-oo 



20 random 



20 random 



20 random 



20 random 



"0.9" 




"0.6" 


0.5 




0.1 


" 1 " 




"2" 


1 




5 


20 




40 


1 




1 










































. 1 . 




.5. 



0.89475' 
0.80000 

■0.19281' 
0.19165 

0.12340 
0.13620 

0.37546 
1.93569 
-1.46461 
0.01287 
0.02212 

1.31000 
0.43157 
0.63367 
0.59941 
0.75423 
0.90423 
1.36573 
4.82393 
2.39867 
4.56890 
5.67535 

'0.75739' 
0.02130 

■ 2.99763 ■ 
3.96642 

79.99969 

0.00236 

0.00060 

0.00136 

0.06457 

3.53081 

26.43154 
0.00449 
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